Ok, I got it.
When I play forward the pause call gets executed and then the timeline stops. So when I go backwards the first thing, that is found in the timeline is the pause call.
In order to fix this, I check in my pauseTL function the direction of the playback and which button is clicked, so that the pause does not get executed on direction change.